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Abstract: Prior to the retirement of the Space 
Shuttle, many exterior repairs on the 
International Space Station (ISS) were carried 
out by shuttle astronauts, trained on the ground 
and flown to the Station to perform these 
specific repairs. With the retirement of the 
shuttle, this is no longer an available option. As 
such, the need for ISS crew members to review 
scenarios while on flight, either for tasks they 
already trained for on the ground or for 
contingency operations has become a very 
critical issue. 

NASA astronauts prepare for Extra- Vehicular 
Activities (EVA) or Spacewalks through 
numerous training media, such as: self-study, 
part task training, underwater training in the 
Neutral Buoyancy Laboratory (NBL), hands-on 
hardware reviews and training at the Virtual 
Reality Laboratory (VRLab). In many 
situations, the time between the last session of a 
training and an EVA task might be 6 to 8 
months. EVA tasks are critical for a mission and 
as time passes the crew members may lose 
proficiency on previously trained tasks and their 
options to refresh or learn a new skill while on 
flight are limited to reading training materials 
and watching videos. In addition, there is an 
increased need for unplanned contingency 


repairs to fix problems arising as the Station 
ages. In order to help the ISS crew members 
maintain EVA proficiency or train for 
contingency repairs during their mission, the 
Johnson Space Center’s VRLab designed an 
immersive ISS Virtual Reality Trainer (VRT). 

The VRT incorporates a unique optical system 
that makes use of the already successful 
Dynamic On-board Ubiquitous Graphics 
(DOUG) software to assist crew members with 
procedure reviews and contingency EVAs while 
on board the Station. 

The need to train and re-train crew members for 
EVAs and contingency scenarios is crucial and 
extremely demanding. ISS crew members are 
now asked to perform EVA tasks for which they 
have not been trained and potentially have never 
seen before. The Virtual Reality Trainer (VRT) 
provides an immersive 3D environment similar 
to the one experienced at the VRLab crew 
training facility at the NASA Johnson Space 
Center. VRT bridges the gap by allowing crew 
members to experience an interactive, 3D 
environment to reinforce skills already learned 
and to explore new work sites and repair 
procedures outside the Station. 
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1. About the VRLab - Ground Based 
Training 

The VRLab is a crew training facility that uses a 
unique custom developed 3D graphics software 
product called Dynamic On-board Ubiquitous 
Graphics (DOUG). The DOUG software 
package is the meat of the virtual reality system, 
as it provides a comprehensive representation of 
the ISS exterior. This software with a suite of 
real-time simulations provides an immersive 
virtual reality environment for high fidelity 
training scenarios for EVA and Robotics tasks. 
With this VRLab system the crew members can 
choreograph and rehearse their on-orbit 
procedures. It is rapidly re-configurable, 
providing an unlimited number of scenarios that 
can be evaluated at a fraction of the cost of other 
more rigid systems. The main training scenarios 
provided by the VRLab are: 

• SAFER (Simplified Aid for EVA rescue) 
Training scenarios. It is a mini-propulsion 


unit that provides contingency maneuvering 
capability for a space-suited crew member 
during extravehicular activity or EVA 
operations. The SAFER provides self-rescue 
capability for an EVA crew member who 
has become inadvertently separated from the 
International Space Station (ISS). 

• The Robotic Workstation (RWS). A 

mockup that represents the Cupola on-board 
ISS, which allows integrated training of 
virtual EVA operations with robotics 
support. The RWS ( see Fig. 1 ) allows crew 
members to practice flying the Space Station 
Remote Manipulator System (SSRMS) and 
is used with and without EVA crew 
members. 

The unique fidelity of the VRLab graphic 
system allows crew members to practice 
flying the SSRMS with realistic lighting 
conditions that is essential for critical 
operations as the capture of visiting vehicles 
such as: Dragon, Cygnus or the H2 Transfer 
Vehicle (HTV) or any future commercial 
vehicles. To simulate the EVA 
environment, the VRLab can support 
operations of the RWS and two EVA crew 
members simultaneously, also allowing the 
crew members to practice the 
communications and teamwork they will use 
during an actual EVA. 

• Mass Handling scenarios. The mass 
handling simulation consists of a robot (or 
two), with six degrees of freedom that 
allows crew members to experience 
handling payloads of various sizes, masses, 
and center-of-gravity locations, in a 
simulated and controlled microgravity 
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environment or zero-g. The two robots 
allow for two crewmembers working 
together to practice joint control or transfer 
of large payloads, (see Fig. 2) 

• EVA Training procedures development. 

The crew members can practice translating 
along the exterior of the Station, just as they 
would on a real EVA. They can examine 
different work sites and translation paths to 
familiarize themselves with where they will 
go for a planned EVA, or where they might 
go for a contingency EVA. This is a unique 
capability since there isn’t any other place in 
the world with the high fidelity up-to-date 
3D model of the ISS. 

To immerse the EVA crew members in the 
virtual environment, the VRLab has two Head- 
Mounted Displays (HMDs). Each HMD has a 
separate display for each eye, thus providing the 
3D virtual environment. Incidentally, the 
current lab HMDs were somewhat a result of the 
VRT design system. Because of the 

development of the VRT, the VRLab staff 
discovered that improved HMD methods and 
materials were becoming available at reasonable 
cost. While the previous generation of HMDs 
in use at the VRLab was very good, the next 
generation developed after the VRT using new 
optics and LCD displays now provides an 
improved immersive experience, with a 

resolution of 1280 x 800 per eye and a 110 
degree horizontal field of view. 

Today, the VRLab is the only training facility 
where crew members can train for operations on 


the entire exterior of the ISS. Given the vast 
size of the Station, there are no full-size replicas 
of the complete spacecraft. DOUG is used in 
the VRLab to provide a 3D, immersive 
environment for the EVA crew members, and it 
is also available to crew members, flight 
controllers and instructors for use on their 
individual computers. The software has been 
integral in the development and execution of 
EVAs and Robotics operations throughout ISS 
assembly. 




Fig. 2. Mass-Handling equipment 
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2. Understanding the training problems on 
flight and the need for a VRT system. 

Even though the current DOUG graphics 
software is available in the ISS on-flight and 
used extensively for crew training prior to their 
flight, an immersive 3D environment on-board 
the Space Station was needed to provide the 
crew members with a similar knowledge 
training experience that they had before their 
mission. An immersive 3D environment allows 
them to get familiarized with the work site 
efficiently and to better understand location, 
orientation, and operations when positioned 
outside the Space Station during an EVA. 

The learning benefits of 3D immersive 
environments are a well-studied topic and have 
identified the positive fast contribution to 
conceptual understanding. The VRT was also 
needed to augment current Station on-board 
trainers, such as the Simplified Aid for EVA 
Rescue On-board Trainer (SAFER SOT), which 
continues training done on the ground but is 
currently very limited due to the reduced field of 
view and lack of a sense of immersion because 
it is run on a regular laptop 2D display. Several 
other virtual reality helmet systems have been 
flown previously to the ISS, but their use has 
been limited to a few unique tasks due to their 
targeted capabilities and the certification and 
time required preparing individual scenarios for 
flight. With the use of the VRT all these 
concerns are taken care off. 

2. Initial VRT design goals and approach 

♦♦♦ To provide an immersive visual experience. 


♦♦♦ To use as much of the existing on-orbit 
equipment and materials as possible. 

❖ Any additional hardware that must be built 
and flown has to be as simple as possible to 
reduce cost and flight certification time. 

❖ Desired properties of the optical system: 

> Utilize as much of the laptop screen as 
possible to maximize resolution. 

> Have good stereo overlap. 

> Provide a wide field of view 

> Place most of the laptop mass over 
center of head to minimize head pan 
inertia. 

> Have a large enough exit pupil to 
accommodate variances in inter- 
pupillary distance without requiring 
optical adjustment. 

> Accommodate add-on prescription 
lenses for those who needed correction. 

One of the key considerations that drove the 
design of the VRT unit was to minimize the 
time and cost of flight certification by utilizing 
many components already on-board the ISS. 
These include: laptops to run the HD 3D visual 
graphic engine; a hand control module (HCM) 
to provide the inputs for flying and menu 
navigation; and a web cam, to provide the 
tracking feedback needed for immersive head 
movement within the virtual environment. There 
was also the need to modify a version of the 
DOUG graphic software to support the new 
VRT mode, and it also had to be easily up 
linkable to the ISS laptop computers. New 
hardware components that needed to be created 
for the VRT included: optical system with lens 
adapter capability, and a viewing hood that 



would fit over the screen of the on-board 
laptops. 

3. Bringing VRT On-board the ISS 

In today’s spaceflight environment, the main 
ISS assembly is complete, and the purpose of 
EVAs has shifted from ISS assembly to smaller, 
less frequent maintenance and repair tasks. As 
such, the crews that perform these EVAs might 
not have the luxury of training the tasks on the 
ground prior to their flight. Virtual reality 
provides the clear advantage of “seeing” and 
training tasks before they must be performed. 

In a critical contingency EVA scenario, where 
hardware must be repaired or replaced in a short 
time frame, the desire to have a Virtual Reality 
Laboratory on-board the ISS became clear. But 
how do you replicate a complicated facility, 
which uses electromagnetic-based motion 
tracking, extensive computational resources, and 
several operators to keep everything running? 
Trade studies were done to survey current 
commercial virtual reality systems, but even 
with recent advances in technology, none of the 
systems provided the desired capability; most 
notably, a wide enough field of view and high 
resolution. 

The VRT ( see Fig. 3) inspiration came in an 
unexpected form, as a laptop screen on the head. 
A prototype was designed and built, looking 
deceptively simple. The VRT consists of a 
hood that attaches to a laptop screen, with an 
integrated optics assembly. The user simply 
straps the laptop to their head - a method only 
suited for the micro-gravity environment on the 
ISS. The modified DOUG software provides 
the visuals, with a mode to split the laptop 


screen’s horizontal view with a normal 
resolution of 1920 x 1200 into two views, left 
and right eye images. The optics assembly 
provides the lenses required to allow the user to 
focus on and converge the two images, thus 
providing a 3D environment with a resolution of 
960 x 1200 per eye and an almost 100 degree 
horizontal field of view. 



Fig. 3. VRT Prototype in use. 


With the visual scene taken care of, the next 
issue became control. As mentioned above, the 
VRLab uses a professional motion tracking 
system, as well as custom gloves to track the 
location of the crew members’ heads and hands 
to provide feedback in the virtual environment. 
Launching such a system to the ISS was simply 
not an option due to cost, spatial constraints, etc. 
Again turning to the equipment already 
available on-board, the engineers at the VRLab 
decided to use a web cam attached to the back 
of the laptop screen to provide head tracking. 
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Software embedded within the DOUG package 
then tracks the optical flow of the scene via the 
web cam to adjust the visuals as the crew 
members moves their head. Another item 
already on-board, the Simplified Aid for EVA 
Rescue (SAFER) hand controller, was chosen to 
maneuver through the visual scene. The 
SAFER is the “jet pack” used by EVA crew 
members for emergency rescue if they were to 
float off of structure untethered. 

4. The VRT components 
Building new components 
• The Optics: 

The optics was designed with two lenses, one is 
a Plano-convex 43mm diameter with 77 mm FL 
and the other is a Plano-concave 50mm 
diameter with -200 mm FL (see Fig. 4.). 

By combining one Plano-concave element 
(50mm dia, -200mm fl) with its planar surface 
parallel to and facing the screen with a Plano- 
convex element (43mm dia, 77mm fl) with its 
planar surface facing the user's eye and angled 
to the screen by 20 degrees we were able to 
achieve a stereo overlap of over 44 degrees and 
create the focal point area for the eye’s pupils to 
see through and converge the image coming 
from the laptop screen for an image exceeding 
90 degrees of horizontal field of view. 



This particular lens arrangement enables the 
needed large stereo overlap due to the angular 
compression that occurs optically as the eye 
peers towards the bridge of the nose and gives a 
combined focal length of 125mm and allowing 
the majority of the laptop mass to sit directly 
above the head minimizing the head's rotational 
inertia in the pan left and right directions. 

The large lenses also allow sufficient eye relief 
to permit the addition of prescription lenses for 
people who needed it and still maintain a 
horizontal field of view exceeding 90 degrees 
and a vertical field of view exceeding 75 
degrees {see Fig. 5) 

This particular configuration provides a 19 mm 
area of focus "sweet spot" to comfortably 
accommodate several inner pupillary sizes 
without needing any mechanical adjustment. In 
order to create this effect, the laptop screen 
should be placed at a distance of approximately 
108 mm (approx. 4.25 inches) away from the 
Plano Concave lens. 
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The adapters are an additional metal component 
created to accommodate the SuperFocus™ 
prescription lenses used by some of the crew 
members on-board the ISS as needed (see Fig. 
6) 
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Fig. 6. Adapter for corrective lenses 


• The Hood: 

The hood is a pyramid shaped base that encloses 
the optics and provides the structure that 
supports the optics assembly at an optimal 
distance and orientation to the laptop screen. 
The hood was constructed using a 3D extrusion 
printer using a special spaceflight rated, flame 
retardant plastic (see Fig. 7) 


• Straps: 

Adjustable straps were added to the final unit to 
securely hold all parts together during flight to 
the ISS and to provide support when attaching 
the hood to the laptop (see Fig. 8) 



• The Mask: 

The VRT used an off-the-shelf safety goggle-style 
mask. A special lip fitting that is built into the 
hood allows the mask to snap into place on the 
hood to provide a comfortable fit for the crew 
member and limit outside light intrusion into the 
VRT system (see Fig. 9) 
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Fig. 9. Comfort Mask 


Using existing components 

• EMU SAFER HCM with Quick Flex 
PCMCIA card: 

The Extra Mobility Unit (EMU) SAFER 
Hand Controller Module (see Fig. 10) is the 
unit that allows the crew member to fly the 
SAFER unit as commanded. It houses both a 
joystick for directional commands and a 
liquid crystal display for monitoring the 
system status from the avionics. The 
joystick is a four axis controller, used either 
for translation or rotation depending on the 
position of a separate mode switch resulting 
in a six degrees of freedom maneuvering: 
+/-X (Roll about X), right/left as +/-Y (Pitch 
about Y) and up/down as -/+ Z (Yaw about 
Z) mapped via a body-centric coordinate 
frame. An additional button called 
Automatic Attitude Hold (AAH) is available 
and can hold any rotation attitude while 
maintaining translational authority. The 
Quick Flex card included with the HCM is a 
technology that provides a dynamic re- 
configurable interface to adapt the hand 
controller’s I2C command interface to the 
Station’s laptops. It uses a custom 


programmed FPGA (Field Programmable 
Gate Array) chip that provides software 
performance at hardware speeds. As all the 
electronic components on-board the ISS are, 
this technology was also radiation tested 
before deployment. 

The HCM provides the interface necessary 
to manipulate the menu entries for the VRT 
since access to the laptop is not possible 
when the VRT is installed. Additionally the 
HCM can also be used to navigate through 
the 3D scene. 
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Fig. 1 0. SAFER Hand Control Module 


• Station Laptop 

The current laptop used on-board the Space 
Station is a Lenovo ThinkPad™ Series T61p 
Notebook with the following components: 

• Processor Intel Core 2 Duo 2.4GHz 

• NVidia Quadro FZ 570M (256MB) 

• Basic Memory: 2GB DDR2 

• Display: 15.4” 1920x1200 

• HDrive 160GB 
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The laptop runs the DOUG 3D graphic package, 
as well as numerous other applications 
commonly used on-board the ISS (see Fig. 11) 



Fig. 11. Lenovo ThinkPad™ Series T61p 


• Web Cam: 

The web cam used on-board the Space Station is 
a Logitech™ QuickCam™ Pro for Notebooks 
with a 960 x 720 HD video format, a 2Meg 
sensor, Carl Zeiss optics for high quality, 
RightLight 2 technology (enhance back light), 
and a clip holder. It records videos at 30 frames 
per sec and works under Windows using a USB 
interface. The web cam provides a real-time 
head tracking capability for the VRT system. 
When connected and interfacing with the 
DOUG software, it detects the physical changes 
in the scene from frame to frame and uses these 
changes to compute pan and tilt movements of 
the camera (and thereby the VRT) using a 
technique called optical-flow. The optical-flow 
implementation into DOUG was done with the 
assistance of the OpenCV library available 
under Windows environments OS (see Fig. 12) 



Fig. 12. ISS Used Webcam 

• Necessary DOUG Software modification: 

The optical characteristics of the lenses, as well 
and the canted installation of the optic results in 
a degree of optical distortion. The 3D graphics 
of the DOUG software was modified to correct 
the optical distortion that occurs especially near 
the center line between the eyes. An inverse 
distortion mapping was applied to the rendered 
images before being displayed to the screen 
resulting in a non-distorted seamless stereo 3D 
experience when viewed through the optical 
system. 

The Optical Distortion Correction was written 
as a plug-in for the DOUG software so that 
future changes to the VRT hardware including 
the optical assembly and any new laptop can be 
accommodated without the need to alter the 
software executable base engine. The DOUG 
plug-in reads an input with the textual 
description of the optical assembly and laptop 
screen from a configuration file. The technique 
used by the plugin to correct for optical 
distortion involves the rendering of the scene 
using a planer projection having a slightly larger 
FOV than needed to an off-screen buffer 
followed by the rendering of a distorted mesh 
textured with the previously rendered scene. 
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The coordinates for the distorted mesh are 
determined at run time by ray tracing using a 
mathematical description of the optical 
assembly from the eye's location in the direction 
of each coordinate “in a uniform mesh” to the 
location where the ray trace intersects the laptop 
screen. As consequence, the final design of the 
unit provided for a much better immersive 
experience than previously used HMD’s for 
ground training at the VRLab. 

5. Final Assembly 

The final VRT assembly in zero-g requires 
attaching the goggle mask to the VRT hood 
first, then attaching the optics insert to the hood, 
adding the insert divide to the inside of the hood 
to physically separate the screen images, and 
strapping the hood to the laptop screen display 
(see Fig. 13). Then the crew member attaches 
the USB Web cam to the back of the laptop for 
tracking and connects the HCM to the Quick 
Flex card on the laptop. At this point the full 
VRT system is functional. 



Fig. 13. The VRT Assembly 


There is an extra frame assembly that can be 
inserted onto the optics block, for use with the 
crew member’s SuperFocus™ prescription 


lenses, if they need corrective optics. This 
assembly is separate and only use if needed ( see 

Fig. 14) 



Fig. 14. A View through The VRT Lenses 


6. On-Board Training with the VRT 

The initial use of the VRT includes EVA 
procedure review and EVA contingency 
operations familiarization, in addition to SAFER 
proficiency training. The crew members have a 
menu option from the VRT (see Fig. 15) to 
select the task to perform such as, playing a pre- 
configured EVA movie generated by trainers on 
the ground prior to an EVA activity or a SAFER 
recue run or simply fly around Station for 
spatial awareness. 



Fig. 15. The VRT Menu 
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Astronauts have been using the DOUG software 
package on-board the Station and Shuttle for 
EVA and Robotics support throughout assembly 
of the ISS. The VRT system (see Fig. 16 & 17) 
takes the capability to the next level by 
providing an immersive 3-D environment on 
flight. Some ideas for future training using the 
VRT include: integrated EVA Robotics training 
through networked laptops on-board, some 
potential augmented reality applications and for 
automated threshold eye perimeter exams to 
measure eye changes that will help in 
quantitating the induced hypertension vision 
alteration problems encountered by crew 
members in and post flight. 

By making use of existing ISS hardware (laptop, 
web cam, hand controller, DOUG software, etc), 
the certification process to fly the hardware was 
simplified and expedited. The VRT hardware 
was manifested and launched on ATV4 June 5, 
2013, and will likely be put to use quickly by 
the ISS crew prior to a set of planned summer 
EVAs. 

7. Conclusion 

The VRT system delivers an efficient way to 
help crew members maintain proficiency and 
review procedures for a previously trained EVA 
task. But most importantly the VRT allows crew 
members to prepare and train for contingency 
EVA scenarios when they occur. The smart 
design of the VRT uses basic components and 
incorporates parts already available on the ISS, 
creating a very useful, versatile, and unique 3D 
training capability on-board the ISS as it is the 
first fully immersive virtual reality system 
available on-board the Space Station. 



Fig. 16. Astronaut Testing The VRT 


Fig. 17. The VRT 
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